Model evaluation
## Figure 5. Sample size estimation at step of neonatal risk. Blue line indicates the LOESS. Red line indicates the modified exponential decay fitting. p(a) is the p-value of a where a is a parameter represents the factor by which the exponential term was scaled in the model and a higher value generally means a quicker initial increase. p(k) is the p-value of k where k is the rate of decay or growth in the exponential term and a positive k implies a decay (as x increases, the impact of increasing y diminishes) while a negative k suggests an erroneous model fit in this context because an increase in x should not lead to a decrease in y. AUC-ROC, the area under curve of receiver operating characteristics; LOESS, locally estimated scatterplot smoothing.
## Figure 6. Sample size estimation at step of BSID-III. Blue line indicates the LOESS. Red line indicates the modified exponential decay fitting. p(a) is the p-value of a where a is a parameter represents the factor by which the exponential term was scaled in the model and a higher value generally means a quicker initial increase. p(k) is the p-value of k where k is the rate of decay or growth in the exponential term and a positive k implies a decay (as x increases, the impact of increasing y diminishes) while a negative k suggests an erroneous model fit in this context because an increase in x should not lead to a decrease in y. AUC-ROC, the area under curve of receiver operating characteristics; LOESS, locally estimated scatterplot smoothing.
## Figure 7. Sample size estimation at step of M-CHAT-R. Blue line indicates the LOESS. Red line indicates the modified exponential decay fitting. p(a) is the p-value of a where a is a parameter represents the factor by which the exponential term was scaled in the model and a higher value generally means a quicker initial increase. p(k) is the p-value of k where k is the rate of decay or growth in the exponential term and a positive k implies a decay (as x increases, the impact of increasing y diminishes) while a negative k suggests an erroneous model fit in this context because an increase in x should not lead to a decrease in y. AUC-ROC, the area under curve of receiver operating characteristics; LOESS, locally estimated scatterplot smoothing.
## Figure 8. Sample size estimation at step of M-CHAT-F. Blue line indicates the LOESS. Red line indicates the modified exponential decay fitting. p(a) is the p-value of a where a is a parameter represents the factor by which the exponential term was scaled in the model and a higher value generally means a quicker initial increase. p(k) is the p-value of k where k is the rate of decay or growth in the exponential term and a positive k implies a decay (as x increases, the impact of increasing y diminishes) while a negative k suggests an erroneous model fit in this context because an increase in x should not lead to a decrease in y. AUC-ROC, the area under curve of receiver operating characteristics; LOESS, locally estimated scatterplot smoothing.
## Figure 9. Sample size estimation at step of M-CHAT-R/F. Blue line indicates the LOESS. Red line indicates the modified exponential decay fitting. p(a) is the p-value of a where a is a parameter represents the factor by which the exponential term was scaled in the model and a higher value generally means a quicker initial increase. p(k) is the p-value of k where k is the rate of decay or growth in the exponential term and a positive k implies a decay (as x increases, the impact of increasing y diminishes) while a negative k suggests an erroneous model fit in this context because an increase in x should not lead to a decrease in y. AUC-ROC, the area under curve of receiver operating characteristics; LOESS, locally estimated scatterplot smoothing.
## Figure 10. Sample size estimation at step of M-CHAT-RF. Blue line indicates the LOESS. Red line indicates the modified exponential decay fitting. p(a) is the p-value of a where a is a parameter represents the factor by which the exponential term was scaled in the model and a higher value generally means a quicker initial increase. p(k) is the p-value of k where k is the rate of decay or growth in the exponential term and a positive k implies a decay (as x increases, the impact of increasing y diminishes) while a negative k suggests an erroneous model fit in this context because an increase in x should not lead to a decrease in y. AUC-ROC, the area under curve of receiver operating characteristics; LOESS, locally estimated scatterplot smoothing.
Table 7. Cost of outcomes.
|
Step
|
How much is the cost of detected/treated disease, e.g, severe risk of
death or severe morbidity despite detection, plus the cost of
intervening?
|
How much is the cost of an unneeded intervention, e.g, invasiveness of
testing, complication risk of treatment?
|
How much is the cost of an undetected disease, e.g, severe risk of death
or severe morbidity?
|
How much is the cost of applying the risk model?
|
|
Universal N
|
30
|
20
|
80
|
2
|
|
Universal NB
|
30
|
20
|
80
|
5
|
|
Universal NBR
|
30
|
20
|
80
|
9
|
|
Universal NBF
|
30
|
20
|
80
|
12
|
|
Universal NBR/F
|
30
|
20
|
80
|
11
|
|
Universal NBRF
|
30
|
20
|
80
|
16
|
|
Universal B
|
30
|
20
|
80
|
3
|
|
Note:
|
|
BSID-III, Bayley scales of infant development version III;
M-CHAT-F, modified checklist for autism in toddlers, follow-up;
M-CHAT-R, modified checklist for autism in toddlers, revised;
M-CHAT-R/F, if M-CHAT-R ≥3 and M-CHAT-F ≥2, then 1, otherwise 0; N,
neonatal risk factors; NB, neonatal risk factors and BSID-III; NBR,
neonatal risk factors, BSID-III, and M-CHAT-R; NBF, neonatal risk
factors, BSID-III, and M-CHAT-F; NBR/F, neonatal risk factors, BSID-III,
and M-CHAT-R/F; NBRF, neonatal risk factors, BSID-III, M-CHAT-R, and
M-CHAT-F.
|
## Figure 11. ASD prediction model calibration plots using neonatal risk based on validation set: (A) RR; (B) NB; (C) KNN; (D) RF; and (E) DNN. Vertical dashed lines indicate the cost-aware threshold. ASD, autism spectrum disorder; DNN, deep neural network; KNN, k-nearest neighbor; NB, naive Bayes; RF, random forest; RR, ridge regression.
## Figure 12. ASD prediction model calibration plots using BSID-III based on validation set: (A) DT; and (B) RF. Vertical dashed lines indicate the cost-aware threshold. ASD, autism spectrum disorder; BSID-III, Bayley scales of infant development version III; DT, decision tree; RF, random forest.
## Figure 13. ASD prediction model calibration plots using M-CHAT-R based on validation set: (A) DT; (B) RF; and (C) GBM. Vertical dashed lines indicate the cost-aware threshold. ASD, autism spectrum disorder; M-CHAT-R, modified checklist for autism in toddlers, revised; DT, decision tree; GBM, gradient boosting machine; RF, random forest.
## Figure 14. ASD prediction model calibration plots using M-CHAT-F based on validation set: (A) DT; (B) RF. Vertical dashed lines indicate the cost-aware threshold. ASD, autism spectrum disorder; M-CHAT-F, modified checklist for autism in toddlers, follow-up; DT, decision tree; RF, random forest.
## Figure 15. ASD prediction model calibration plots using M-CHAT-R/F based on validation set: (A) DNN; (B) DT; (C) RF. Vertical dashed lines indicate the cost-aware threshold. ASD, autism spectrum disorder; M-CHAT-F, modified checklist for autism in toddlers, follow-up; M-CHAT-R, modified checklist for autism in toddlers, revised; M-CHAT-R/F, if M-CHAT-R ≥3 and M-CHAT-F ≥2, then 1, otherwise 0; DNN, deep neural network; DT, decision tree; RF, random forest.
## Figure 16. ASD prediction model calibration plots using M-CHAT-RF based on validation set: (A) DT; (B) RF. Vertical dashed lines indicate the cost-aware threshold. ASD, autism spectrum disorder; M-CHAT-F, modified checklist for autism in toddlers, follow-up; M-CHAT-R, modified checklist for autism in toddlers, revised; M-CHAT-RF, both M-CHAT-R and M-CHAT-F were used; DNN, deep neural network; DT, decision tree; RF, random forest.
## Figure 17. SHAP beeswarm plot of best prediction model using neonatal risk factors: (A) RR; and (B) KNN. Interval estimate of SHAP values are shown for higher feature values (>= average). CI, confidence interval; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 18. SHAP beeswarm plot of best prediction model using BSID-III: (A) DT; and (B) RF. Interval estimate of SHAP values are shown for higher feature values (>= average). BSID-III, Bayley scales of infant development version III; CI, confidence interval; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 19. SHAP beeswarm plot of best prediction model using M-CHAT-R: (A) RF; and (B) GBM. Interval estimate of SHAP values are shown for higher feature values (>= average). CI, confidence interval; M-CHAT-R, modified checklist for autism in toddlers, revised; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 20. SHAP beeswarm plot of best prediction model using M-CHAT-F: (A) RF. Interval estimate of SHAP values are shown for higher feature values (>= average). CI, confidence interval; M-CHAT-F, modified checklist for autism in toddlers, follow-up; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 21. SHAP beeswarm plot of best prediction model using M-CHAT-R/F: (A) DT; (B) RF; and (C) DNN. Interval estimate of SHAP values are shown for higher feature values (>= average). CI, confidence interval; M-CHAT-F, modified checklist for autism in toddlers, follow-up; M-CHAT-R, modified checklist for autism in toddlers, revised; M-CHAT-R/F, if M-CHAT-R ≥3 and M-CHAT-F ≥2, then 1, otherwise 0; DNN, deep neural network; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 22. SHAP beeswarm plot of best prediction model using M-CHAT-RF: (A) RF. Interval estimate of SHAP values are shown for higher feature values (>= average). CI, confidence interval; M-CHAT-F, modified checklist for autism in toddlers, follow-up; M-CHAT-R, modified checklist for autism in toddlers, revised; M-CHAT-RF, both M-CHAT-R and M-CHAT-F were used; DNN, deep neural network; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 23. SHAP beeswarm plot of best prediction model using only BSID-III: (A) DT; (B) RF; and (C) GBM. Interval estimate of SHAP values are shown for higher feature values (>= average). CI, confidence interval; M-CHAT-F, modified checklist for autism in toddlers, follow-up; M-CHAT-R, modified checklist for autism in toddlers, revised; M-CHAT-RF, both M-CHAT-R and M-CHAT-F were used; DNN, deep neural network; OR, odds ratio; SHAP, Shapley additive explanation.
## Figure 24. Feature impact alignment with multivariate regression analysis. *, aligned according to positive-to-negative ratio; †, aligned according to clinician assessment; ‡, unclear alignment according to clinician assessment; Bayley scales of infant development version III; M-CHAT-F, modified checklist for autism in toddlers, follow-up; M-CHAT-R, modified checklist for autism in toddlers, revised; M-CHAT-R/F, if M-CHAT-R ≥3 and M-CHAT-F ≥2, then 1, otherwise 0; M-CHAT-RF, both M-CHAT-R and M-CHAT-F were used; N, neonatal risk factors; NB, neonatal risk factors and BSID-III; NBR, neonatal risk factors, BSID-III, and M-CHAT-R; NBF, neonatal risk factors, BSID-III, and M-CHAT-F; NBR/F, neonatal risk factors, BSID-III, and M-CHAT-R/F; NBRF, neonatal risk factors, BSID-III, and M-CHAT-RF.
## Best model: KNN
## Figure 25. ASD prediction model decision (A) and ROC (B) curves using neonatal risk based on validation set. Performances using the selected threshold are shown by dashed line in (A) and points in (B). ASD, autism spectrum disorder; DNN, deep neural network; KNN, k-nearest neighbor; NB, naive Bayes; RF, random forest; ROC, receiver operating characteristics; RR, ridge regression.
## Best model: GBM
## Figure 26. ASD prediction model decision (A) and ROC (B) curves using M-CHAT-R based on validation set. Performances using the selected threshold are shown by dashed line in (A) and points in (B). ASD, autism spectrum disorder; M-CHAT-R, modified checklist for autism in toddlers, revised; DT, decision tree; GBM, gradient boosting machine; RF, random forest; ROC, receiver operating characteristics.
## Best model: GBM
## Figure 27. ASD prediction model decision (A) and ROC (B) curves using only BSID-III based on validation set. Performances using the selected threshold are shown by dashed line in (A) and points in (B). ASD, autism spectrum disorder; BSID-III, Bayley scales of infant development version III; GBM, gradient boosting machine; ROC, receiver operating characteristics.
Table 8. Validation results (95% CI).
|
Step
|
Algorithm
|
AUC-ROC
|
F1
|
TPR
|
PPV
|
TNR
|
NPV
|
|
Universal N
|
KNN
|
0.682 (0.655, 0.709)
|
0.35 (0.32, 0.38)
|
58 (52, 64)
|
25 (23, 28)
|
68 (66, 71)
|
90 (88, 91)
|
|
Universal NBR
|
GBM
|
0.749 (0.726, 0.771)
|
0.37 (0.33, 0.41)
|
49 (44, 53)
|
31 (27, 35)
|
79 (77, 81)
|
89 (88, 90)
|
|
Universal B
|
GBM
|
0.637 (0.604, 0.67)
|
0.32 (0.28, 0.36)
|
39 (34, 44)
|
29 (25, 33)
|
81 (79, 83)
|
87 (86, 89)
|
|
Note:
|
|
AUC-ROC, area under curve of receiver operating
characteristics; BSID-III, Bayley scales of infant development version
III; CI, confidence interval; F1, 2 x precision x recall / (precision +
recall); FNR, false negative rate or specificity; FPR, false positive
rate or sensitivity/recall; M-CHAT-F, modified checklist for autism in
toddlers, follow-up; M-CHAT-R, modified checklist for autism in
toddlers, revised; N, neonatal risk factors; NBR, neonatal risk factors,
BSID-III, and M-CHAT-R; NPV, negative predictive value or precision;
PPV, positive predictive value or precision; TNR, true negative rate or
specificity; TPR, true positive rate or sensitivity/recall.
|
## Figure 28. ASD prediction step comparison. AUC-ROC, area under curve of receiver operating characteristics; ASD, autism spectrum disorder; BSID-III, Bayley scales of infant development version III; CI, confidence interval; F1, 2 x precision x recall / (precision + recall); FNR, false negative rate or specificity; FPR, false positive rate or sensitivity/recall; M-CHAT-R, modified checklist for autism in toddlers, revised; N, neonatal risk factors; NBR, neonatal risk factors, BSID-III, and M-CHAT-R; NPV, negative predictive value or precision; PPV, positive predictive value or precision; TNR, true negative rate or specificity; TPR, true positive rate or sensitivity/recall.
